Cooling a data center

ABSTRACT

Techniques for cooling a data center include circulating a first cooling medium to cool a plurality of rack-mounted computers; circulating a second cooling medium to cool the plurality of rack-mounted computers; determining that a first portion of the plurality of rack-mounted computers is operating at a power usage above a threshold power usage; adjusting at least one of a flow rate of the first or second cooling mediums to cool a second portion of the plurality of rack-mounted computers; and rerouting a portion at least one the first or second cooling mediums to cool the first portion of the plurality of rack-mounted computers.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. §119 to U.S.Provisional Patent Application Ser. No. 61/731,934, entitled “Cooling aData Center,” filed on Nov. 30, 2012, the entire contents of which areincorporated herein by reference.

TECHNICAL BACKGROUND

This disclosure relates to managing performance of one or more computersthat operate in a data center.

BACKGROUND

Computer users often focus on the speed of computer microprocessors,e.g., megahertz and gigahertz. Many forget that this speed often comeswith a cost—higher power consumption. For one or two home PCs, thisextra power may be negligible when compared to the cost of running themany other electrical appliances in a home. But in data centerapplications, where thousands of microprocessors may be operated,electrical power requirements can be very important.

Power consumption is also, in effect, a double whammy. Not only must adata center operator pay for electricity to operate its many computers,but the operator must also pay to cool the computers. That is because,by simple laws of physics, all the power has to go somewhere, and thatsomewhere is, in the end, conversion into heat. A pair ofmicroprocessors mounted on a single motherboard can draw hundreds ofwatts or more of power. Multiply that figure by several thousand, ortens of thousands, to account for the many computers in a large datacenter, and one can readily appreciate the amount of heat that can begenerated. It is much like having a room filled with thousands ofburning floodlights. The effects of power consumed by the critical loadin the data center are often compounded when one incorporates all of theancillary equipment required to support the critical load.

Thus, the cost of removing all of the heat can also be a major cost ofoperating large data centers. That cost typically involves the use ofeven more energy, in the form of electricity and natural gas, to operatechillers, condensers, pumps, fans, cooling towers, and other relatedcomponents. Heat removal can also be important because, althoughmicroprocessors may not be as sensitive to heat as are people, increasesin temperature can cause great increases in microprocessor errors andfailures. In sum, a data center requires a large amount of electricityto power the critical load, and even more electricity to cool the load.

SUMMARY

In a general implementation, techniques for cooling a data centerincludes circulating a cooling airflow to cool a plurality of computingdevices supported in racks in a data center; monitoring power usage ofthe plurality of computing devices; determining that a first portion ofthe plurality of computing devices are operating at a power usage abovea threshold power usage; and based on the determination, decreasing aflow rate of a portion of a cooling liquid in the data center that iscirculated to cool a second portion of the plurality of computingdevices that are operating at a power usage below the threshold powerusage; increasing a portion of the cooling airflow circulated to coolthe second portion of the plurality of computing devices such that arate of heat removal from the second portion of the plurality ofcomputing devices remains substantially constant; and increasing a flowrate of a portion of the cooling liquid in the data center that iscirculated to cool the first portion of the plurality of computingdevices such that a rate of heat removal from the first portion of theplurality of computing devices increases.

In a first aspect combinable with the general implementation, monitoringpower usage of the plurality of computing devices includes monitoring autilization of each of the plurality of computing devices.

In a second aspect combinable with any of the previous aspects, thethreshold power usage corresponds to an adjustable threshold utilizationthat is a particular percentage of a maximum utilization.

In a third aspect combinable with any of the previous aspects, theadjustable threshold utilization is about 70% of maximum utilization.

In a fourth aspect combinable with any of the previous aspects,utilization includes CPU utilization of the plurality of computingdevices.

A fifth aspect combinable with any of the previous aspects furtherincludes rerouting at least some of the portion of a cooling liquid inthe data center that is circulated to cool the second portion of theplurality of computing devices to increase the flow rate of the portionof the cooling liquid in the data center that is circulated to cool thefirst portion of the plurality of computing devices.

A sixth aspect combinable with any of the previous aspects furtherincludes monitoring a temperature at or near the first portion of theplurality of computing devices.

A seventh aspect combinable with any of the previous aspects furtherincludes subsequent to increasing flow rate of a portion of the coolingliquid in the data center that is circulated to cool the first portionof the plurality of computing devices, determining that the temperatureremains substantially constant or that the temperature rises at or nearthe first portion of the plurality of computing devices.

An eighth aspect combinable with any of the previous aspects furtherincludes based on the determination that the temperature remainssubstantially constant or that the temperature rises at or near thefirst portion of the plurality of computing devices, increasing aportion of the cooling airflow circulated to cool the first portion ofthe plurality of computing devices such that the rate of heat removalfrom the first portion of the plurality of computing devices furtherincreases.

A ninth aspect combinable with any of the previous aspects furtherincludes determining that the first portion of the plurality ofcomputing devices are operating at a power usage at or below thethreshold power usage for a predetermined time duration, or that thefirst portion of the plurality of computing devices are operating at apower usage at a particular setpoint below the threshold power usage.

A tenth aspect combinable with any of the previous aspects furtherincludes based on the determination, decreasing the flow rate of theportion of the cooling liquid in the data center that is circulated tocool the first portion of the plurality of computing devices.

An eleventh aspect combinable with any of the previous aspects furtherincludes modulating an airflow circulated to cool the first portion ofthe plurality of computing devices to maintain the rate of heat removalfrom the first portion of the plurality of computing devices.

In a twelfth aspect combinable with any of the previous aspects, thedata center includes a cooling capacity that is less than a cooling loadrequired to cool all of the plurality of computing devices operating ata maximum power draw.

A thirteenth aspect combinable with any of the previous aspects furtherincludes deploying an additional plurality of computing devicessupported in racks in the data center.

In another general implementation, a data center cooling managementsystem includes a plurality of rack-mounted computers; one or morecooling modules positioned near the plurality of rack-mounted computers;and a control system including one or more sensors and one or more flowcontrol devices. The control system is operable to perform operationsincluding controlling one or more cooling modules to circulate a coolingairflow to cool the plurality of rack-mounted computers; monitoring,with the one or more sensors, power usage of the plurality ofrack-mounted computers; determining, with the one or more sensors, thata first portion of the plurality of rack-mounted computers are operatingat a power usage above a threshold power usage; and based on thedetermination, controlling the one or more flow control devices todecrease a flow rate of a portion of a cooling liquid in the data centerthat is circulated to cool a second portion of the plurality ofrack-mounted computers that are operating at a power usage below thethreshold power usage; controlling the one or more cooling modules toincrease a portion of the cooling airflow circulated to cool the secondportion of the plurality of rack-mounted computers such that a rate ofheat removal from the second portion of the plurality of rack-mountedcomputers remains substantially constant; and controlling the one ormore flow control devices to increase a flow rate of a portion of thecooling liquid in the data center that is circulated to cool the firstportion of the plurality of rack-mounted computers such that a rate ofheat removal from the first portion of the plurality of rack-mountedcomputers increases.

In a first aspect combinable with the general implementation, monitoringpower usage of the plurality of rack-mounted computers includesmonitoring a utilization of each of the plurality of rack-mountedcomputers.

In a second aspect combinable with any of the previous aspects, thethreshold power usage corresponds to an adjustable threshold utilizationthat is a particular percentage of a maximum utilization.

In a third aspect combinable with any of the previous aspects, theadjustable threshold utilization is about 70% of maximum utilization.

In a fourth aspect combinable with any of the previous aspects, theoperations further include controlling the one or more flow controldevices to circulate at least some of the portion of the cooling liquidin the data center that is circulated to cool the second portion of theplurality of rack-mounted computers to increase the flow rate of theportion of the cooling liquid in the data center that is circulated tocool the first portion of the plurality of rack-mounted computers.

In a fifth aspect combinable with any of the previous aspects, theoperations further include monitoring, with the one or more sensors, atemperature at or near the first portion of the plurality ofrack-mounted computers.

In a sixth aspect combinable with any of the previous aspects, theoperations further include subsequent to increasing flow rate of aportion of the cooling liquid in the data center that is circulated tocool the first portion of the plurality of rack-mounted computers,determining that the temperature remains substantially constant or thatthe temperature rises at or near the first portion of the plurality ofrack-mounted computers.

In a seventh aspect combinable with any of the previous aspects, theoperations further include based on the determination that thetemperature remains substantially constant or that the temperature risesat or near the first portion of the plurality of rack-mounted computers,controlling the one or more cooling modules to increase a portion of thecooling airflow circulated to cool the first portion of the plurality ofrack-mounted computers such that a rate of heat removal from the firstportion of the plurality of rack-mounted computers further increases.

In an eighth aspect combinable with any of the previous aspects, theoperations further include determining that the first portion of theplurality of rack-mounted computers are operating at a power usage at orbelow the threshold power usage for a predetermined time duration, orthat the first portion of the plurality of rack-mounted computers areoperating at a power usage at a particular setpoint below the thresholdpower usage.

In a ninth aspect combinable with any of the previous aspects, theoperations further include based on the determination, controlling theone or more flow control devices to decrease the flow rate of theportion of the cooling liquid in the data center that is circulated tocool the first portion of the plurality of rack-mounted computers.

In a tenth aspect combinable with any of the previous aspects, theoperations further include controlling the one or more cooling devicesto modulate an airflow circulated to cool the first portion of theplurality of rack-mounted computers to maintain the rate of heat removalfrom the first portion of the plurality of rack-mounted computers.

In an eleventh aspect combinable with any of the previous aspects, theone or more flow control devices include one or more pumps; one or morecontrol valves; or one or more variable frequency motor controllers.

In a twelfth aspect combinable with any of the previous aspects furtherincludes one or more cooling plants that include a cooling capacity thatis less than a cooling load to cool all of the plurality of rack-mountedcomputers operating at a maximum power draw.

In another general implementation, a method for cooling a data centerincludes circulating a first cooling medium to cool a plurality ofrack-mounted computers; circulating a second cooling medium to cool theplurality of rack-mounted computers; determining that a first portion ofthe plurality of rack-mounted computers is operating at a power usageabove a threshold power usage; adjusting at least one of a flow rate ofthe first or second cooling mediums to cool a second portion of theplurality of rack-mounted computers; and rerouting a portion at leastone of a flow rate of the first or second cooling mediums to cool thefirst portion of the plurality of rack-mounted computers.

A first aspect combinable with the general implementation furtherincludes determining that the second portion of the plurality ofrack-mounted computers is operating at a power usage below a thresholdpower usage.

In a second aspect combinable with any of the previous aspects, thefirst cooling medium is a cooling airflow and the second cooling mediumis a cooling liquid.

In a third aspect combinable with any of the previous aspects, adjustingat least one of a flow rate of the first or second cooling mediums tocool a second portion of the plurality of rack-mounted computersincludes increasing a flow rate of the cooling airflow to cool thesecond portion of the plurality of rack-mounted computers; anddecreasing a flow rate of the cooling liquid to cool the second portionof the plurality of rack-mounted computers.

In a fourth aspect combinable with any of the previous aspects,circulating a second cooling medium to cool the plurality ofrack-mounted computers includes circulating, to the data center, acooling liquid flow from one or more central plants that include acooling capacity less than a maximum cooling capacity required to coolthe plurality of rack-mounted computers operating at a maximum powerload.

Various implementations of techniques for cooling a data center mayinclude one or more of the following features. For example, a datacenter with excess airflow capacity (e.g., the capability to circulatemore airflow than would ever be needed in particular locations) mayincrease airflow rates to cool certain groups of computers or servers sothat cooling liquid may be rerouted to cool other groups of computers orservers. As data center cooling capacity may not have excess coolingliquid available, rerouting the available cooling liquid may moreefficiently cool computers that are operating at a maximum or higherthan normal power load. In some aspects, a vast majority of computers orservers may operate most efficiently even while the data center has thecapability to cool the hardest working computers or servers. As anotherexample, such techniques may allow for construction of a data center(and more particularly, one or more central cooling plants) to bereduced in cooling capacity while still being capable of meetingyear-round cooling needs. As another example, a data center with excesscooling liquid capacity (e.g., the capability to circulate more coolingliquid than would ever be needed in particular locations) may increasecooling liquid flow rates to cool certain groups of computers or serversso that cooling airflow may be rerouted to cool other groups ofcomputers or servers.

These general and specific aspects may be implemented using a device,system or method, or any combinations of devices, systems, or methods.For example, a system of one or more computers can be configured toperform particular actions by virtue of having software, firmware,hardware, or a combination of them installed on the system that inoperation causes or cause the system to perform the actions. One or morecomputer programs can be configured to perform particular actions byvirtue of including instructions that, when executed by data processingapparatus, cause the apparatus to perform the actions. The details ofone or more implementations are set forth in the accompanying drawingsand the description below. Other features, objects, and advantages willbe apparent from the description and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIGS. 1A-1C show various views of an example tray in a rack-mountcomputer system;

FIGS. 1D-1F show various views of an example tray in a rack-mountcomputer system, having dual-zone power supply ventilation;

FIG. 1G shows a plan view of an example tray in a rack-mount computersystem, having dual-zone adjustable power supply ventilation;

FIGS. 2A-2B illustrate an example method for managing cooling ofcomputing devices in a data center;

FIG. 3 shows a plan view of two rows in a computer data center withcooling units arranged between racks situated in the rows;

FIGS. 4A-4B show plan and sectional views, respectively, of a modulardata center system;

FIG. 5 illustrates a schematic diagram showing an example system forcooling a computer data center; and

FIG. 6 illustrates a distribution function of machine utilization in adata center that includes multiple rack-mounted computers.

DETAILED DESCRIPTION

FIG. 1A shows a plan view of a tray 10 in a rack-mount computer system,while FIG. 1B shows a front view, and FIG. 1C shows a side view, of thetray 10 in FIG. 1A. The term “tray” is not limited to any particulararrangement, but instead includes any arrangement of computer-relatedcomponents coupled together to serve a particular purpose, such as on amotherboard. Trays may be generally mounted parallel to other trays in ahorizontal or vertical stack, so as to permit denser packing than wouldotherwise be possible with computers having free-standing housings andother components. The term “blade” may also be employed to refer to suchapparatuses, though that term too should not be limited to a particulararrangement. Trays may be implemented in particular configurations,including as computer servers, switches, e.g., electrical and optical,routers, drives or groups of drives, and other computing-relateddevices. In general, the trays in a system take a standardized physicaland electrical form to be easily interchangeable from one location inthe system to another, but the trays may take other appropriate forms.

Various implementations of techniques for cooling a data center mayinclude one or more of the following features. For example, a datacenter with excess airflow capacity (e.g., the capability to circulatemore airflow than would ever be needed in particular locations) mayincrease airflow rates to cool certain groups of computers or servers sothat cooling liquid may be rerouted to cool other groups of computers orservers. As data center cooling capacity may not have excess coolingliquid available, rerouting the available cooling liquid may moreefficiently cool computers that are operating at a maximum or higherthan normal power load. In some aspects, a vast majority of computers orservers may operate most efficiently even while the data center has thecapability to cool the hardest working computers or servers. As anotherexample, such techniques may allow for construction of a data center(and more particularly, one or more central cooling plants) to bereduced in cooling capacity while still being capable of meetingyear-round cooling needs. As another example, a data center with excesscooling liquid capacity (e.g., the capability to circulate more coolingliquid than would ever be needed in particular locations) may increasecooling liquid flow rates to cool certain groups of computers or serversso that cooling airflow may be rerouted to cool other groups ofcomputers or servers.

In some general implementations, and as described more fully below, tray10 (and many multiples of tray 10) may be installed in a data center andoperated to execute and/or manage one or more software applications. Inso operating, the trays 10 may generate significant amounts of heat thatmust be removed by, for instance, cooling modules (e.g., fan coil units)that are positioned near the trays 10. In some aspects, such coolingmodules may capture an airflow that is circulated over the trays 10 andheated by the components of the tray 10. Once captured (e.g., in a warmair plenum), the air may be cooled (e.g., by cooling coils) andrecirculated to again flow over the trays 10.

In some implementations, placement of the cooling modules may bedetermined by, for instance, a predicted or calculated power draw by thetrays 10. For example, more cooling modules may be placed near trays 10that are predicted to have high usage or utilization and, therefore,high heat generation. Less cooling modules may be placed near trays 10that are predicted to have low usage or utilization and, therefore, lowheat generation. The applications running on the trays 10, and the powerusage of the trays 10, may vary significantly from the predicatedoperational constraints.

To better dissipate heat load from trays 10 that are operating at ahigher than expected usage, cooling modules may be physically moved.This, however, may be impractical. As another solution, excess airflowcapacity may be utilized. But due to the nature of air circulation(e.g., cost per cooling capacity unit, horsepower requirements, andotherwise), this too may be impractical. Modulating and rerouting acooling liquid, e.g., chilled water, to pinpoint locations of trays 10that need more cooling may be more efficient (e.g., due to relativecosts of pumping liquid vs. air, relative costs of installing piping vs.ductwork, and otherwise). Thus, the capability to “move cooling liquid”from location to location in the data center may be advantageous asexplained herein.

In general, the tray 10 may include a standard circuit board 12 on whicha variety of components are mounted. The board 12 may be arranged sothat air enters at its front edge (to the left in the figure), is routedover a number of heat generating components on the board 12, and isdrawn through a power supply 14 and fan 16 before being exhausted fromthe tray 10. The fan 16 may also be arranged to push air through thepower supply 14. In addition, the fan 16 may be located in otherpositions relative at other positions along the back edge of the trayand at locations away from a back edge of the tray 10. The power supply14 may likewise be positioned at other locations and need not be joinedto the fan 16.

In this arrangement, the heat from power supply 14 may be picked upafter the heat from other components on the board 12 is picked up by theair flow. In this manner, the speed of fan 16 may be controlled tomaintain a set temperature for the air exiting the board 12, or fortemperatures at other points on the tray 10. For example, a thermocoupleor other sort of temperature sensor may be placed in the air flow, suchas upstream of the power supply 14 or downstream of the fan 16, and thefan speed may be modulated to maintain a set temperature. Thetemperature of the exiting air may also be highly elevated compared tosystems that do not control airflow in this manner. It may be moreefficient to cool this air than it would be to cool air that does nothave such an elevated temperature.

Air may be routed over board 12 by walls 26 a, 26 b, 26 c. Wall 26 a mayblock one side of board 12, and may funnel air toward openings in powersupply 14. Where the walls 26 a, 26 c do not taper, the air mayotherwise be directed to the fan 16. Wall 26 c may block one side ofboard 12, so as to prevent air from moving directly from the workspaceinto an area behind tray 10, e.g., to the right in the figure. Forexample, a plenum may be provided behind multiple boards in the form ofan open wall into which the boards may be placed, or in the form of awall having multiple openings into which fans may be slid. In certainimplementations, fully blocking or sealing of such a plenum may not benecessary, such as when the pressure difference between the plenum andthe workspace is minimal.

Wall 26 b separates one portion of tray 10 from another. In particular,wall 26 b separates the portion of tray 10 containing heat generatingcomponents, such as microprocessors 21 a, 21 b, from components thatgenerate substantially less heat, such as hard drives 18 a, 18 b. Inmaking such a separation, wall 26 b substantially blocks airflow overthe components that generate less heat, and increases airflow over theheat generating components. In addition, wall 26 b is arranged to routeairflow into openings in power supply 14. Although not pictured, wall 26b may block areas on tray 10 but may provide that each blocked area,e.g., the area on each side of wall 26 b, may still be in fluidcommunication with fan 16. For example, fan 16 may be designed to haveopenings that will lie on each side of wall 26 b, and the openings maybe sized or otherwise tuned so as to provide for relative levels of airflow on the opposing sides of the wall 26 b. The “tuning” of the airflow may be made to match the relative thermal load of components oneach side of wall 26 b, so that more air flows on the side of wall 26 bhaving the most thermal load, or that otherwise requires more cooling.

Board 12 may hold a variety of components needed in a computer system.As shown, board 12 holds a dual processor computer system that usesprocessor 21 a and processor 21 b connected to a bank of memory 24. Thememory 24 may be in the form, for example, of a number of single in-linememory modules (SIMMs), dual in-line memory module (DIMMs), or otherappropriate form. Other components of the computer system, such as chipsets and other chips, have been omitted for clarity in the figure, andmay be selected and arranged in any appropriate manner.

Board 12 may also be provided with connections to other devices. Networkjack 22, such as in the form of an RJ-45 jack or an optical networkingconnection, may provide a network connection for tray 10. Otherconnections may also be provided, such as other optical networkingconnections, video output connections, and input connections such askeyboard or pointing device connections (not shown).

Impingement fans 20 a, 20 b may be mounted above each microprocessor 21a, 21 b, to blow air downward on the microprocessors 21 a, 21 b. In thismanner, impingement fans 20 a, 20 b may reduce boundary layer effectsthat may otherwise create additional heat buildup on microprocessors 21a, 21 b. As a result, lateral airflow across tray 10 can be reduced evenfurther, while still adequately controlling the temperature rise to themicroprocessors 21 a, 21 b.

In the illustrated embodiment, actuators 11 are shown communicablycoupled to impingement fans 20 a and 20 b, as well as fan 16. In someimplementations, the actuators 11 may control the fans, e.g., speed,based on one or more inputs, such as, for example, operatingtemperature, microprocessor frequency, or other input. For example, asexplained more fully in FIG. 2, the actuators 11 may be controlled basedon a control output based on thermal margin of a microprocessor, e.g.,microprocessors 21 a and 21 b.

Other heat relief mechanisms may also, or alternatively, be provided formicroprocessors 21 a, 21 b. For example, one or more heat sinks may beprovided, such as in the form of certain finned, thermally conductivestructures. The heat sinks may be directly connected to microprocessors21 a, 21 b, or may be located to the sides of microprocessors 21 a, 21b, and may be attached by heat pipes to plates mounted to the top ofmicroprocessors 21 a, 21 b. Thermally conductive grease or paste may beprovided between the tops of microprocessors 21 a, 21 b, and any heatsinks to improve heat flow out of microprocessors 21 a, 21 b.

In operation, tray 10 may be mounted flat horizontally in a server racksuch as by sliding tray 10 into the rack from the rack front, and over apair of rails in the rack on opposed sides of the tray 10—much likesliding a lunch tray into a cafeteria rack, or a tray into a bread rack.Tray 10 may alternatively be mounted vertically, such as in a bank oftrays mounted at one level in a rack. The front of the rack may be keptopen to permit easy access to, and replacement of, trays and to permitfor air to flow over the tray 10 from a workspace where technicians orother professionals operating a data center may be located. In thiscontext, the term workspace is intended to refer to areas in whichtechnicians or others may normally be located to work on computers in adata center.

After sliding a tray 10 into a rack, a technician may connect a tray toappropriate services, such as a power supply connection, batteryback-up, and a network connection. The tray 10 may then be activated, orbooted up, and may be communicated with by other components in thesystem.

Although tray 10 is shown in the figures to include a multi-processorcomputer system, other arrangements may be appropriate for other trays.For example, tray 10 may include only hard drives and associatedcircuitry if the purpose of the tray is for storage. Also, tray 10 maybe provided with expansion cards such as by use of a riser modulemounted transversely to the board 12. Although particular forms of tray10 may be provided, certain advantages may be achieved in appropriatecircumstances by the use of common trays across a rack or multipleracks. In particular, great efficiencies may be gained by standardizingon one or a small handful of trays so as to make interaction betweentrays more predictable, and to lower the need to track and store manydifferent kinds of trays.

A data center may be made up of numerous trays (hundreds or thousands),each mounted in one of numerous racks. For example, several dozen traysmay be mounted in a single rack within a space, with approximatelyseveral inches between each tray. As explained in more detail below,each of the trays in a rack may back up to a warm air plenum thatreceives exhaust air from the trays and routes that air to a coolingunit that may re-circulate the air into the workspace in front of theracks.

Trays may also be packaged in groups. For example, two stacked trays maybe matched as a pair, with one fan 16 serving both trays (not shown).Specifically, the fan 16 may be approximately double the height anddiameter of a single tray unit, and may extend from the lower tray in apair up to the top of the upper tray in a pair. By such an arrangement,the slowest turning portions of the fan, in the fan center, will be nearthe board of the top tray, where less airflow will normally occurbecause of boundary layer effects. The larger and faster moving portionsof the fan 16 will be located nearer to the free areas of each tray 10so as to more efficiently move air over the trays and through therespective power supplies more freely. In addition, a double-height fanmay be able to move more air than can a single-height fan, at lowerrotation speeds. As a result, a fan in such an arrangement may produceless noise, or noise at a more tolerable frequency, than could a smallerfan. Parallel fans may also be used to increase flow, and serial fansmay be used to increase pressure, where appropriate.

Fan 16 may be controlled to maintain a constant temperature for airexiting fan 16 or at another point. By locating fan 16 downstream ofpower supply 14, and power supply 14 downstream of the other componentsof tray 10, the arrangement may maximize the heat rise across tray 10,while still maintaining adequately low temperatures for heat-sensitivecomponents mounted to board 12, such as microprocessors 21 a, 21 b.Also, the power supply 14 may be less sensitive to higher temperaturesthan are other components, and so may be best located at the end of theair flow, where the temperatures are highest.

Although many applications seek to substantially increase airflow acrossheat generating components so as to increase the rate of heatdissipation from the components, the arrangement pictured here allowsairflow across tray 10 to be slowed substantially to increase thetemperature rise across tray 10. Increasing the temperature risedecreases the mass flow rate, and can make cooling across the entiresystem more efficient.

In particular, when the temperature of the warm exiting air isincreased, the difference in temperature between the warm air and anycooling water entering a cooling coil to cool the warm air, alsoincreases. The ease of heat transfer is generally directly proportionalto this difference in temperature. Also, when the difference intemperature is relatively small, increasing the difference by only oneor two degrees can produce a substantial increase in the amount of heatexchange between the warm air and the cooling water. As a result, asystem run at higher exhaust temperatures from board 12 can providesubstantial advantages in efficiency, and lower energy consumption.

In certain implementations, the temperature rise across tray 10 may beapproximately 20° C. As one example, air may enter the space above board12 from a workspace at 25° C., and may exit fan 16 at 45° C. Theentering temperature may also be about 21-30° C. (70-86° F.), and theexiting temperature 40-50° C. (104-122° F.). The 45° C. exhausttemperature or other temperature may be selected as a maximumtemperature for which the components in tray 10 can be maintainedwithout significant errors or breakdowns, or a safe temperature ofoperation. The 25° C. entering temperature or other temperature may be atemperature determined to create a comfortable or tolerable temperaturein the workspace in a data center. The entering temperature may also belinked to a maximum allowable temperature, such as a federal or stateOSHA-mandated maximum temperature. The entering temperature could beapproximately 40° Celsius, which matches certain limits established bybodies governing workplace safety.

In other implementations, air may enter the space above board 12 at atemperature of 50° C., where appropriate thermal removal mechanisms ormethods are provided for the components on board 12. For example,conductive and liquid-cooled components may be placed in contact withmicroprocessors 21 a, 21 b to increase the rate of heat dissipation fromthose components. Where a higher input temperature is selected, thetemperature difference across tray 10 will generally be lower than if alower input temperature is selected. However, heat will be easier toremove from such heated air when it passes through a cooling coil.Higher temperatures for expected breakdowns include components thattolerate case temperatures of 85 degrees Celsius. In addition, the exitair temperature from tray 10 may be as high as 75 degrees Celsius. Anoutput temperature may be most easily controlled by locating atemperature sensor at the actual output (or aiming it at the actualoutput. Such an output temperature may also be controlled or maintainedwithin an acceptable temperature range by placing a temperature sensorat a location away from the output, but where the difference intemperature is adequately predictable.

In the front view of FIG. 1B, one can see power supply 14 located at theback of tray 10, and perforated to permit the flow of air through powersupply 14. In addition, one can see hard drive 18 a located in an areawalled off from the heat generating components of tray 10 by wall 26 b.As noted above, the power supply 14 could also be situated so as toreceive air leaving two different zones on tray 10, with the powersupply 14 or other components tuned to maintain certain relative airflow rates from each side.

The side view of FIG. 1C shows more clearly the relationship of theimpingement fans 20 a, 20 b and microprocessors 21 a, 21 b. The fans 20a, 20 b are shown schematically for clarity. Air is pulled through thetops of fans 20 a, 20 b, and driven down against the top ofmicroprocessors 21 a, 21 b. This process breaks up layers of warm airthat may otherwise form above microprocessors 21 a, 21 b.

As noted above, other techniques for spot removal of heat fromcomponents such as microprocessors 21 a, 21 b may also be employed. Asone example, heat sinks may be attached on top of or to the side ofmicroprocessors 21 a, 21 b, and may be cooled by circulating air or aliquid, such as water or fluorinert liquid, or oils. Liquid supply andreturn tubes may be provided down each rack, with taps at which toconnect pipes for cooling particular components. Circulation of liquidto the components may be driven by pressure created centrally in thesystem, e.g., from natural tap water pressure or large pumps, or bysmall pumps local to a particular tray 10. For example, smallperistaltic, centrifugal, vane or gear-rotor pumps may be provided witheach tray to create liquid circulation for the tray 10.

Alternatively, a portion of a rack or a component associated with a rackmay be cooled, such as by passing liquid through passages in thecomponent. Heat sinks for each heat generating component may then becoupled physically to the cooled component in the rack so as to drawheat out of the components on the tray 10 and into the rack. As oneexample, a vertical runner on the rack may be provided with clamps intowhich heat pipes attached to heat-generating components on tray 12 arereceived, so that the heat pipes may pull heat away from thosecomponents and into the runner. The runner may further include fluidpassages to carry cooling fluid. Thus, the runner will be kept cool, andwill draw heat by conduction from the heat-generating components.

FIG. 1D shows a plan view of a tray in a rack-mount computer system,having dual-zone power supply ventilation. FIG. 1E shows a front view ofthe tray in FIG. 1D. FIG. 1F shows a side view of the tray in FIG. 1D.The general arrangement of components on the tray 12 here is similar tothat in FIGS. 1A-1C, although the particular arrangement and layout ofcomponents is not generally critical. However, in these figures, thewall 26 b has its rear edge pulled forward from the back wall of thetray 12. Also, the power supply 14 has two areas of openings—one on itsfront edge, as can be seen in FIG. 1E, and one on its side edge, as canbe seen in FIG. 1F. The openings on the front edge generally provideventilation for the hot side of the tray 12, while those on the sideedge provide ventilations for the cool side of the tray 12.

The openings may be sized or otherwise organized to provide particularapproximate levels of ventilation to each side of the tray 12. As can beseen in FIGS. 1E and 1F, the front edge of the power supply 1 has moreholes than does the edge; in addition, the air flow from the front edgeis straight, while air coming in through the side edge needs to curve.As a result, the front edge will provide a higher level of ventilationthan will the side edge, and will thus be able to carry away the higherlevel of heat generated on the hot side of tray 12. The amount of aircarried on a hot side might also be lower than on a cool side, such aswhere equipment requirements force the cool side to stay at a lowtemperature. In other words, in setting flow rates for each portion oftray 12, both heat generation and desired operating temperature may betaken into account.

FIG. 1G shows a plan view of a tray in a rack-mount computer system,having dual-zone adjustable power supply ventilation. Here, the wall 26b is positioned to direct a certain amount of ventilating air from eachside of wall 26 b. The wall 26 b may be positioned on tray 12 at anappropriate position, and its terminal end may be made adjustablethrough pivoting or other mechanisms, so as to permit on-site adjustmentof air flow.

In addition, gate 27 may be provided over a front surface of powersupply 14 to provide adjustment to the size of openings on the frontsurface via openings in the gate 27 that form an interference patternwith openings on power supply 27 (much like the openings on certainspice containers). The interference pattern may be different for eachside of tray 12, so that moving the gate 27 causes a greater effect onthe airflow for one side of tray 12 than it does for the other side oftray 12.

Temperature-dependent mechanisms may also be provided to control theflow of air through power supply 14. For example, polymer or metallicmaterials that change shape with temperature may be used to formopenings that close as their temperature falls—thereby driving back upthe exit temperature of air from a particular portion of tray 12. As oneexample, the materials may produce a form of stoma that opens andcloses. Also, mechanisms such as temperature-controlled louvers, or atemperature-controlled actuator on gate 27 may be used to controlairflow over board 12. Such air control mechanisms may also be locatedoff of tray 12. For example, a wall perforated by temperature dependentstoma may be placed behind a number of racks filled with trays, and maythereby control the exit temperature for all of the racks in aconvenient manner. In such a situation, as in others discussed herein,fan 16 may be eliminated from tray 12, and a central ventilation systemmay pull air through the various trays and racks.

FIGS. 2A-2B illustrate an example method 200 for managing cooling ofcomputing devices in a data center. Method 200 may be implemented, forexample, by or with any appropriate cooling system for a data center,such as, the cooling systems, modules, and apparatus described herein(e.g., with reference to FIGS. 3, 4A-4B, and/or 5). In an exampleimplementation, a control system (e.g., control system 503) mayimplement one or more steps of method 200.

In step 202, a cooling airflow is circulated to cool rack-mountedcomputers in a data center. The computers may be mounted in racks ortrays, such as those illustrated in FIGS. 1A-1G, and in racks as shown,for instance, in FIGS. 3, 4A-4B, or 5, to name a few implementations.The airflow may be circulated, for example, by fans that are mounted onthe trays or racks, or by fans of cooling modules positioned betweenracks (e.g., FIG. 3), or by other fans in a data center cooling system.In some implementations, the cooling airflow may be determined accordingto an amount of heat generated by the computers during operation. Thisamount of heat is dissipated, for example, through the cooling airflowand, eventually, to ambient air, such as through a cooling liquidcirculated through cooling coils arranged to capture the airflow heatedby the computers. In some instances, one or more cooling plants aresized to remove the computer-generated heat (e.g., by chillers, coolingtowers, evaporative coolers, or a combination thereof). In someimplementations, although all of the computers in the data center maygenerate a particular amount of heat to be removed when each computer isoperating at its highest power draw (e.g., highest utilization), the oneor more cooling plants may not have enough cooling capacity to dissipatethat amount of heat. For instance, the cooling plant(s) may be sized todissipate an amount of heat generated by the rack-mounted computers (andother loads) when each computer is operating at an average power draw orutilization (or some combination of computers operating at particularpower loads, and still other computers operating a different powerloads). In some aspects, additional computers may be added to the datacenter (e.g., in one, two, or more deployments beyond an initial numberof computers installed in the data center) even though the one or morecooling plants may not have enough cooling capacity to dissipate theamount of heat generated by the initial number of computers in the datacenter.

Turning to FIG. 6, this figure illustrates graph 600 that shows adistribution function of machine utilization in a data center thatincludes multiple rack-mounted computers. As illustrated, the graph 600includes a y-axis that represents a percentage of machines (e.g.,computers, servers, CPUs) that are operating at a utilization less thana particular threshold value 606. Here, the threshold value 606 is setat about 0.7, or 70%, utilization of a maximum utilization. The x-axis604 represents machine (e.g., CPU) utilization percentage of maximum(e.g., 1.0 or 100%). Graph 600 shows two distribution curves that, forexample, may represent two different types of cooling modules deployedin a data center to cool the rack-mounted computers. For example, insome aspects, one or both of the distribution curves 608 and 610 mayrepresent variations of cooling modules as shown in FIG. 3 (e.g.,cooling modules 302). In another aspect, one of the distribution curves608 and 610 may represent variations of cooling systems as shown inFIGS. 4A-4B and 5, as well as FIG. 3.

As illustrated, the distribution curves 608 and 610 both illustrate adistribution in which about 10% of the machines are operating above thethreshold 606. Thus, as shown, the other 90% of the machines may beadjusted to operate at a more efficient state (e.g., by increasingcooling to such machines). Further, if the 10% of machines that areoperating at or above the threshold 606, additional cooling (e.g.,cooling liquid) may be re-routed from some of the cooling modules thatserve the 90% of the machines to cooling modules that serve the 10% ofthe machines. This may be because, as described herein, the data centermay have a cooling liquid (e.g., chilled water) supply volume that isless than what may be required to cool all the machines operating at100% utilization, but may have an excess cooling airflow capacity in thedata center.

As illustrated, the difference in the distribution curves 608 and 610 isthat, for the distribution curve 608, a higher percentage of machinesare operating at a utilization less than the value of the x-axis 604.Thus, the curve 608 may show more machines are operating at moreefficient (e.g., lower utilization) operating points as compared to thecurve 610.

In step 204, the power usage of the rack-mounted computers is monitored(e.g., for a rack, a row of racks, each individual computer, or someother measure of computers in the data center). In some aspects,processor frequency or computer utilization (e.g., CPU utilization) ismonitored, for example, as a proxy for power load.

In step 206, a determination is made whether a first portion of therack-mounted computers are operating at a power usage above a thresholdpower usage. If no, then method 200 may, in the illustrated example,return to step 200. If yes, then in step 208, a cooling liquid flow(e.g., chilled or cool water, glycol, or other liquid) that is used tocool a second portion of the rack-mounted computers that are operatingbelow the threshold power usage is decreased. In some aspects,decreasing the flow may include, for instance, modulating one or morecontrol valves, turning off one or more pumps, modulating one or morevariable frequency (speed) drives that control pumps used to circulatethe cooling liquid, or other actions. In some aspects, the coolingliquid flow that is decreased is routed to cooling modules (e.g., fancoil units) that are positioned to cool the rack-mounted computer.

In step 210, a cooling airflow to cool the second portion of computersis increased such that a rate of heat removal from the second portion ofcomputers remains substantially constant. In some aspects, increasingthe airflow may include turning on additional rack-mounted (or traymounted) fans, speeding up such fans, modulating fans that are part ofcooling modules positioned to cool the computers, or other action. Heatremoval may remain substantially constant because, as cooling liquidflow rate decreases, the flow of liquid to cool the second portion ofcomputers has less cooling capacity. To account for this loss, morecooling airflow may be circulated to keep a particular temperaturedifference (or leaving air temperature) across (or from) the coolingmodule.

In step 212, a cooling liquid circulated to cool the first portion isincreased in flow rate, such that a rate of heat removal from the firstportion of computers increases (or does not decrease). In some aspects,even though portions of the rack-mounted computers, such as the firstportion, may be operating above a threshold, many other computers (suchas the second portion) may be operating below the threshold power usageor utilization. Cooling modules positioned to cool the second portion ofcomputers may, therefore, have excess capacity (e.g., excess airflow).Such excess airflow, however, may not be easily redirected (e.g., acrossthe data center) to provide extra cooling for the first portion ofcomputers (e.g., computers operating above the threshold). Moreover,there may not be an excess of cooling liquid (e.g., chilled water) inthe data center.

In step 214, some of the cooling liquid used to cool the second portionof the plurality is rerouted to increase the flow rate of the coolingliquid to cool the first portion of computers. Thus, by decreasing aflow of cooling liquid from computers that are operating below athreshold power usage (and making up the difference by increasing anairflow or just allowing such computers to heat up), more cooling liquidcan be circulated to computers that are operating above the threshold.

Turning now to FIG. 2B, in step 216, a temperature at or near the firstportion of rack-mounted computers is monitored (e.g., by temperaturesensors of a control system). In some aspects, the temperature may besensed on the racks that support the second portion of computers. Inother aspects, the temperature may be sensed in a warm air plenum thatis positioned on a back side of the racks (e.g., in between the racksand cooling coils of one or more cooling modules).

In step 218, a determination is made whether the sensed temperatureremains substantially constant or rises at or near the first portion ofrack-mounted computers. If the temperature falls, it may be continuallymonitored in step 216. If the temperature does rise or remainsubstantially constant, then, in step 220, the cooling airflowcirculated to cool the first portion of rack-mounted computers may beincreased such that the rate of heat removal from the first portion ofrack-mounted computers further increases.

In step 222, a determination is made whether the first portion of therack-mounted computers are operating at a power usage at or below thethreshold for a predetermined time duration (e.g., if the power usagehas decreased below the threshold for a specific time period). If no,then a determination is made, in step 224, whether the first portion ofthe rack-mounted computers are operating at a power usage at aparticular setpoint below the threshold (e.g., if the power usage hasdecreased much below the threshold for even a short period of time).

If the determination in either of steps 222 or 224 is yes, then, in step226, the flow rate of the cooling liquid that is circulated to cool thefirst portion of rack-mounted computers is decreased. In step 228, anairflow circulated to the first portion of the rack-mounted computers ismodulated to maintain the rate of heat removal from the first portion ofthe rack-mounted computers. For example, as the flow rate of coolingfluid is decreased (e.g., because the computers have decreased in powerusage or utilization), the airflow may also be decreased.

FIG. 3 shows a plan view of two rows 362 and 364, respectively, in acomputer data center 300 with cooling units 302 arranged between rackssituated in the rows. In some implementations, the data center 300 mayimplement one or more of the computer performance management controlschemes discussed herein. In general, this figure illustrates certainlevels of density and flexibility that may be achieved with structureslike those discussed above. Each of the rows 362, 364 is made up of arow of cooling units 302 sandwiched by two rows 330 of computing racks331. In some implementations, a row may also be provided with a singlerow of computer racks, such as by pushing the cooling units up against awall of a data center, providing blanking panels all across one side ofa cooling unit row, or by providing cooling units that only haveopenings on one side.

Each of the rows of computer racks and rows of cooling units in each ofrows 362, 364 may have a certain cooling unit density. In particular, acertain number of such computing or cooling units may repeat over acertain length of a row such as over 100 feet. Or, expressed in anotherway, each of the cooling units may repeat once every X feet in a row.

In this example, each of the rows is approximately 40 feet long. Each ofthe three-bay racks is approximately six feet long. And each of thecooling units is slightly longer than each of the racks. Thus, forexample, if each rack were exactly six feet long and all of the rackswere adjoining, the rack cooling units would repeat every six feet. As aresult, the racks could be said to have a six-foot “pitch.”

As can be seen, the pitch for the cooling unit rows is different in row362 than in row 364. Row 362 contains five cooling units 302, while row364 contains six cooling units 302. Thus, if one assumes that the totallength of each row is 42 feet, then the pitch of cooling units in row364 would be 7 feet, 42/6, and the pitch of cooling units in row 362would be 8.4 feet, 42/5.

The pitch of the cooling units and of the computer racks may differ, andthe respective lengths of the two kinds of apparatuses may differ,because warm air is able to flow up and down the rows 330. Thus, forexample, a bay or rack may exhaust warm air in an area in which there isno cooling unit to receive it. But that warm air may be drawn laterallydown the row and into an adjacent module, where it is cooled andcirculated back into the work space, such as aisle 332.

Row 362 may receive less cooling air than would row 364. However, it ispossible that row 362 needs less cooling, so that the particular numberof cooling units in each row has been calculated to match the expectedcooling requirements. For example, row 362 may be outfitted with traysholding new, low-power microprocessors; row 362 may contain more storagetrays, which are generally lower power than processor trays, and fewerprocessor trays; or row 362 may generally be assigned lesscomputationally intensive work than is row 364.

In addition, the two rows 362 and 364 may both have had an equal numberof cooling units at one time, but then an operator of the data centermay have determined that row 362 did not need as many modules to operateeffectively. As a result, the operator may have removed one of themodules so that it could be used elsewhere.

The particular density of cooling units that is required may be computedby first computing the heat output of computer racks on both sides of anentire row. The amount of cooling provided by one cooling unit may beknown, and may be divided into the total computed heat load and roundedup to get the number of required cooling units. Those cooling units maythen be spaced along a row so as to be as equally spaced as practical,or to match the location of the heat load as closely as practical, suchas where certain computer racks in the row generate more heat than doothers. Also, as explained in more detail below, the row of coolingunits may be aligned with rows of support columns in a facility, and thecooling units may be spaced along the row so as to avoid hitting anycolumns.

Where there is space between cooling units, a blanking panel 368 may beused to block the space so that air from the warm air capture plenumdoes not escape upward into the work space. The panel 368 may simplytake the form of a paired set of sheet metal sheets that slide relativeto each other along slots 370 in one of the sheets, and can be fixed inlocation by tightening a connector onto the slots.

FIG. 3 also shows a rack 331 a being removed for maintenance orreplacement. The rack 331 a may be mounted on caster wheels so that oneof technicians 372 could pull it forward into aisle 332 and then roll itaway. In the figure, a blanking panel 374 has been placed over anopening left by the removal of rack 331 a to prevent air from the workspace from being pulled into the warm air capture plenum, or to preventwarm air from the plenum from mixing into the work space. The blankingpanel 374 may be a solid panel, a flexible sheet, or may take any otherappropriate form.

In one implementation, a space may be laid out with cooling unitsmounted side-to-side for maximum density, but half of the cooling unitsmay be omitted upon installation, e.g., so that there is 50% coverage.Such an arrangement may adequately match the cooling unit capacity,e.g., about four racks per cooling unit, where the racks areapproximately the same length as the cooling units and mountedback-to-back on the cooling units, to the heat load of the racks. Wherehigher powered racks are used, the cooling units may be moved closer toeach other to adapt for the higher heat load, e.g., if rack spacing islimited by maximum cable lengths, or the racks may be spaced from eachother sufficiently so that the cooling units do not need to be moved. Inthis way, flexibility may be achieved by altering the rack pitch or byaltering the cooling unit pitch.

FIGS. 4A-4B show plan and sectional views, respectively, of a modulardata center system. In some implementations, one of more data processingcenters 400 may implement one or more of the computer performancemanagement control schemes discussed herein. The system may include oneof more data processing centers 400 in shipping containers 402. Althoughnot shown to scale in the figure, each shipping container 402 may beapproximately 40 feet along, 8 feet wide, and 9.5 feet tall, e.g., a1AAA shipping container. In other implementations, the shippingcontainer can have different dimensions, e.g., the shipping containercan be a 1CC shipping container. Such containers may be employed as partof a rapid deployment data center.

Each container 402 includes side panels that are designed to be removed.Each container 402 also includes equipment designed to enable thecontainer to be fully connected with an adjacent container. Suchconnections enable common access to the equipment in multiple attachedcontainers, a common environment, and an enclosed environmental space.

Each container 402 may include vestibules 404 and 406 at each end of therelevant container 402. When multiple containers are connected to eachother, these vestibules provide access across the containers. One ormore patch panels or other networking components to permit for theoperation of data processing center 400 may also be located investibules 404 and 406. In addition, vestibules 404 and 406 may containconnections and controls for the shipping container. For example,cooling pipes, e.g., from heat exchangers that provide cooling waterthat has been cooled by water supplied from a source of cooling such asa cooling tower, may pass through the end walls of a container, and maybe provided with shut-off valves in the vestibules 404 and 406 to permitfor simplified connection of the data center to, for example, coolingwater piping. Also, switching equipment may be located in the vestibules404 and 406 to control equipment in the container 402. The vestibules404 and 406 may also include connections and controls for attachingmultiple containers 402 together. As one example, the connections mayenable a single external cooling water connection, while the internalcooling lines are attached together via connections accessible investibules 404 and 406. Other utilities may be linkable in the samemanner.

Central workspaces 408 may be defined down the middle of shippingcontainers 402 as aisles in which engineers, technicians, and otherworkers may move when maintaining and monitoring the data processingcenter 400. For example, workspaces 408 may provide room in whichworkers may remove trays from racks and replace them with new trays. Ingeneral, each workspace 408 is sized to permit for free movement byworkers and to permit manipulation of the various components in dataprocessing center 400, including providing space to slide trays out oftheir racks comfortably. When multiple containers 402 are joined, theworkspaces 408 may generally be accessed from vestibules 404 and 406.

A number of racks such as rack 419 may be arrayed on each side of aworkspace 408. Each rack may hold several dozen trays, like tray 420, onwhich are mounted various computer components. The trays may simply beheld into position on ledges in each rack, and may be stacked one overthe other. Individual trays may be removed from a rack, or an entirerack may be moved into a workspace 408.

The racks may be arranged into a number of bays such as bay 418. In thefigure, each bay includes six racks and may be approximately 8 feetwide. The container 402 includes four bays on each side of eachworkspace 408. Space may be provided between adjacent bays to provideaccess between the bays, and to provide space for mounting controls orother components associated with each bay. Various other arrangementsfor racks and bays may also be employed as appropriate.

Warm air plenums 410 and 414 are located behind the racks and along theexterior walls of the shipping container 402. A larger joint warm airplenum 412 is formed where the two shipping containers are connected.The warm air plenums receive air that has been pulled over trays, suchas tray 420, from workspace 408. The air movement may be created by fanslocated on the racks, in the floor, or in other locations. For example,if fans are located on the trays and each of the fans on the associatedtrays is controlled to exhaust air at one temperature, such as 40° C.,42.5° C., 45° C., 47.5° C., 50° C., 52.5° C., 55° C., or 57.5° C., theair in plenums 410, 412, and 414 will generally be a single temperatureor almost a single temperature. As a result, there may be little needfor blending or mixing of air in warm air plenums 410, 412, and 414.Alternatively, if fans in the floor are used, there will be a greaterdegree temperature variation from air flowing over the racks, andgreater degree of mingling of air in the plenums 410, 412, and 414 tohelp maintain a consistent temperature profile.

FIG. 4B shows a sectional view of the data center from FIG. 4A. Thisfigure more clearly shows the relationship and airflow betweenworkspaces 408 and warm air plenums 410, 412, and 414. In particular,air is drawn across trays, such as tray 420, by fans at the back of thetrays 419. Although individual fans associated with single trays or asmall number of trays, other arrangements of fans may also be provided.For example, larger fans or blowers, may be provided to serve more thanone tray, to serve a rack or group or racks, or may be installed in thefloor, in the plenum space, or other location.

Air may be drawn out of warm air plenums 410, 412, and 414 by fans 422,424, 426, and 428. Fans 422, 424, 426, and 428 may take various forms.In one exemplary implementation, the may be in the form of a number ofsquirrel cage fans. The fans may be located along the length ofcontainer 402, and below the racks, as shown in FIG. 4B. A number offans may be associated with each fan motor, so that groups of fans maybe swapped out if there is a failure of a motor or fan. Although notshown, each fan or a group of fans may be communicably coupled to acontroller, e.g., an actuator, so as to control a speed of the fan(s).

An elevated floor 430 may be provided at or near the bottom of theracks, on which workers in workspaces 408 may stand. The elevated floor430 may be formed of a perforated material, of a grating, or of meshmaterial that permits air from fans 422 and 424 to flow into workspaces408. Various forms of industrial flooring and platform materials may beused to produce a suitable floor that has low pressure losses.

Fans 422, 424, 426, and 428 may blow heated air from warm air plenums410, 412, and 414 through cooling coils 462, 464, 466, and 468. Thecooling coils may be sized using well known techniques, and may bestandard coils in the form of air-to-water heat exchangers providing alow air pressure drop, such as a 0.5 inch pressure drop. Cooling watermay be provided to the cooling coils at a temperature, for example, of10, 15, or 20 degrees Celsius, and may be returned from cooling coils ata temperature of 20, 25, 30, 35, or 40 degrees Celsius. In otherimplementations, cooling water may be supplied at 15, 10, or 20 degreesCelsius, and may be returned at temperatures of about 25 degreesCelsius, 30 degrees Celsius, 35 degrees Celsius, 45 degrees Celsius, 50degrees Celsius, or higher temperatures. The position of the fans 422,424, 426, and 428 and the coils 462, 464, 466, and 468 may also bereversed, so as to give easier access to the fans for maintenance andreplacement. In such an arrangement, the fans will draw air through thecooling coils.

The particular supply and return temperatures may be selected as aparameter or boundary condition for the system, or may be a variablethat depends on other parameters of the system. Likewise, the supply orreturn temperature may be monitored and used as a control input for thesystem, or may be left to range freely as a dependent variable of otherparameters in the system. For example, the temperature in workspaces 408may be set, as may the temperature of air entering plenums 410, 412, and414. The flow rate of cooling water and/or the temperature of thecooling water may then vary based on the amount of cooling needed tomaintain those set temperatures.

The particular positioning of components in shipping container 402 maybe altered to meet particular needs. For example, the location of fansand cooling coils may be changed to provide for fewer changes in thedirection of airflow or to grant easier access for maintenance, such asto clean or replace coils or fan motors. Appropriate techniques may alsobe used to lessen the noise created in workspace 408 by fans. Forexample, placing coils in front of the fans may help to deaden noisecreated by the fans. Also, selection of materials and the layout ofcomponents may be made to lessen pressure drop so as to permit forquieter operation of fans, including by permitting lower rotationalspeeds of the fans. The equipment may also be positioned to enable easyaccess to connect one container to another, and also to disconnect themlater. Utilities and other services may also be positioned to enableeasy access and connections between containers 402.

Airflow in warm air plenums 410, 412, and 414 may be controlled viapressure sensors. For example, the fans may be controlled so that thepressure in warm air plenums is roughly equal to the pressure inworkspaces 408. Taps for the pressure sensors may be placed in anyappropriate location for approximating a pressure differential acrossthe trays 420. For example, one tap may be placed in a central portionof plenum 412, while another may be placed on the workspace 408 side ofa wall separating plenum 412 from workspace 408. For example the sensorsmay be operated in a conventional manner with a control system tocontrol the operation of fans 422, 424, 426, and 428. One sensor may beprovided in each plenum, and the fans for a plenum or a portion of aplenum may be ganged on a single control point.

For operations, the system may better isolate problems in one area fromother components. For instance, if a particular rack has trays that areoutputting very warm air, such action will not affect a pressure sensorin the plenum, even if the fans on the rack are running at high speed,because pressure differences quickly dissipate, and the air will bedrawn out of the plenum with other cooler air. The air of varyingtemperature will ultimately be mixed adequately in the plenum, in aworkspace, or in an area between the plenum and the workspace.

FIG. 5 illustrates a schematic diagram showing an example system 500 forcooling a computer data center 501. In some implementations, the examplesystem 500 may implement one or more of the computer performancemanagement control schemes discussed herein. The system 500 generallyincludes an air handling unit (including, e.g., fan 510 and coolingcoils 512 a, 512 b) in the data center 501 for transferring heat fromthe data center's air to cooling water, a heat exchanger 522 forremoving heat from the cooling water and passing it to cooling towerwater, and a cooling tower 518 to pass the accumulated heat to theambient air through evaporation and cooling of the cooling tower water.In general operation, the system 500 may be run from the coolingtower/heat exchanger/cooling coil system, though a powered refrigerationsystem such as a chiller may be provided for peak loads, such as whenthe outdoor ambient dew point is very high and the cooling tower cannotprovide sufficient cooling alone. As explained below, control parametersfor the system may also be set so as to avoid most or any need for theuse of chillers or other such items.

The temperatures of each portion of the system 500 are selected to berelatively high, so as to permit more efficient operation of the system500, than if the temperatures were lower. For example, relatively highair temperatures in the system, e.g., air entering a cooling coil over110° F. (43.3° C.) and exiting temperature above 70° F. (43.34° C.), mayin turn permit for relatively high cooling water temperatures, e.g.,water entering a cooling coil around 68° F. (20° C.) and exiting around104° F. (40° C.), because the amount of heat that can be taken out ofthe air is generally proportional to the difference in temperaturebetween the water and the air. If the difference can be kept at anacceptable level, where the temperatures are high enough thatevaporative cooling, e.g., cooling through a cooling tower, withoutfurther cooling via chiller, is sufficient, the relatively highelectrical cost of operating a chiller (or many chillers) may beavoided.

High system temperatures may be particularly advantageous in certainimplementations when hybrid cooling towers are used. Such hybrid coolingtowers combine the functionality of an ordinary cooling tower with awater-to-water heat exchanger. Sufficiently high chosen temperaturesetpoints may allow the hybrid tower to provide substantial coolingcapacity, even when operating in a water-to-air mode without utilitywater. As a result, a hybrid cooling tower may be used to providecooling capacity to a facility relatively quickly, even before utilitywater may be obtained in large volumes. The capacity of the coolingtower may be directly related to the difference in the temperature ofthe water within it to the ambient outside air.

When the difference in temperatures is not very large, a change of onlya few degrees can bring substantial gains in efficiency. For example,where the cooling water enters at 68° F. (20° C.), by heating air to113° F. (45° C.) rather than 104° F. (40° C.), the temperaturedifference is increased from 68° F. to 77° F. (36° C. to 45° C.)—whichmay result in an increase in heat flow of 25 percent. The actualdifference will vary slightly, as the entering conditions for air andwater are not the only conditions (because the air cools as it passesthrough a cooling cool, and the water warms); this example, however,indicates how the difference in temperature can affect efficiency of asystem.

Use of elevated temperatures in a system may also prevent air in oraround the system from falling below its liquid saturation point, i.e.,its dew point, and condensing. This may, in certain circumstances,provide benefits both in efficiency and in operations of the system.Efficiency benefits may be obtained because creating condensationrequires much more energy than simply cooling air, so that systemscreating condensation may use a large amount of electricity or otherenergy. Improvements in operations of the system may occur because, ifpipes in the system carry water that is below the saturation temperatureof the air around the pipes, condensation might form on the pipes. Thatcondensation can damage the pipes or equipment in the conditioned space,cause mold, and cause water to pool on the floor, and can require theinstallation of insulation on the pipes (to stop the condensation).

In the system shown in FIG. 5, use of elevated temperatures maysubstantially reduce, or almost entirely eliminate, the need forenergy-intensive cooling components such as chillers and the like, evenwhere the heat load in the data center 501 is very high. As a result,system 500 may be operated at a lower operating cost than couldotherwise be achieved. In addition, lower capital costs may be required,because fans, coils, heat exchangers, and cooling towers are relativelybasic and inexpensive components. In addition, by operating with ahigher temperature difference between cooled air and cooling water, lessvolume of cooling water is needed, thus reducing the size and cost ofpiping, and the cost to operate pumps and other such components.

In addition, those components are often very standardized, so that theiracquisition costs are lower, and they are more easily located,particularly in developing countries and remote areas where it may bebeneficial to place a data center 501. Use of system 500 in remote areasand other areas with limited access to electrical power is also helpedby the fact that system 500 may be operated using less electrical power.As a result, such a system can be located near lower-power electricalsub-stations and the like. As discussed more completely below,lower-powered systems may also be amenable to being implemented asself-powered systems using energy sources such as solar, wind,natural-gas powered turbines, fuel cells, and the like.

Referring now to FIG. 5, there is shown a data center 501 in sectionalview, which as shown, is a building that houses a large number ofcomputers or similar heat-generating electronic components. A workspace506 is defined around the computers, which are arranged in a number ofparallel rows and mounted in vertical racks, such as racks 502 a, 502 b.The racks may include pairs of vertical rails to which are attachedpaired mounting brackets (not shown). Trays containing computers, suchas standard circuit boards in the form of motherboards, may be placed onthe mounting brackets.

In one example, the mounting brackets may be angled rails welded orotherwise adhered to vertical rails in the frame of a rack, and traysmay include motherboards that are slid into place on top of thebrackets, similar to the manner in which food trays are slid ontostorage racks in a cafeteria, or bread trays are slid into bread racks.The trays may be spaced closely together to maximize the number of traysin a data center, but sufficiently far apart to contain all thecomponents on the trays and to permit air circulation between the trays.

Other arrangements may also be used. For example, trays may be mountedvertically in groups, such as in the form of computer blades. The traysmay simply rest in a rack and be electrically connected after they areslid into place, or they may be provided with mechanisms, such aselectrical traces along one edge, that create electrical and dataconnections when they are slid into place.

Air may circulate from workspace 506 across the trays and into warm-airplenums 504 a, 504 b behind the trays. The air may be drawn into thetrays by fans mounted at the back of the trays (not shown). The fans maybe programmed or otherwise configured to maintain a set exhausttemperature for the air into the warm air plenum, and may also beprogrammed or otherwise configured to maintain a particular temperaturerise across the trays. Where the temperature of the air in the workspace 506 is known, controlling the exhaust temperature also indirectlycontrols the temperature rise. The work space 506 may, in certaincircumstances, be referenced as a “cold aisle,” and the plenums 504 a,504 b as “warm aisles.”

The temperature rise can be large. For example, the work space 506temperature may be about 77° F. (25° C.) and the exhaust temperatureinto the warm-air plenums 504 a, 504 b may be set to 113° F. (45° C.),for a 36° F. (20° C.)) rise in temperature. The exhaust temperature mayalso be as much as 212° F. (100° C.) where the heat generating equipmentcan operate at such elevated temperature. For example, the temperatureof the air exiting the equipment and entering the warm-air plenum may be118.4, 122, 129.2, 136.4, 143.6, 150.8, 158, 165, 172.4, 179.6, 186.8,194, 201, or 208.4° F. (48, 50, 54, 58, 62, 66, 70, 74, 78, 82, 86, 90,94, or 98° C.). Such a high exhaust temperature generally runs contraryto teachings that cooling of heat-generating electronic equipment isbest conducted by washing the equipment with large amounts offast-moving, cool air. Such a cool-air approach does cool the equipment,but it also uses lots of energy.

Cooling of particular electronic equipment, such as microprocessors, maybe improved even where the flow of air across the trays is slow, byattaching impingement fans to the tops of the microprocessors or otherparticularly warm components, or by providing heat pipes and relatedheat exchangers for such components.

The heated air may be routed upward into a ceiling area, or attic 505,or into a raised floor or basement, or other appropriate space, and maybe gathered there by air handling units that include, for example, fan510, which may include, for example, one or more centrifugal fansappropriately sized for the task. The fan 510 may then deliver the airback into a plenum 508 located adjacent to the workspace 506. The plenum508 may be simply a bay-sized area in the middle of a row of racks, thathas been left empty of racks, and that has been isolated from anywarm-air plenums on either side of it, and from cold-air work space 506on its other sides. Alternatively, air may be cooled by coils defining aborder of warm-air plenums 504 a, 504 b and expelled directly intoworkspace 506, such as at the tops of warm-air plenums 504 a, 504 b.

Cooling coils 512 a, 512 b may be located on opposed sides of the plenumapproximately flush with the fronts of the racks. (The racks in the samerow as the plenum 508, coming in and out of the page in the figure, arenot shown.) The coils may have a large surface area and be very thin soas to present a low pressure drop to the system 500. In this way,slower, smaller, and quieter fans may be used to drive air through thesystem. Protective structures such as louvers or wire mesh may be placedin front of the coils 512 a, 512 b to prevent them from being damaged.

In operation, fan 510 pushes air down into plenum 508, causing increasedpressure in plenum 508 to push air out through cooling coils 512 a, 512b. As the air passes through the coils 512 a, 512 b, its heat istransferred into the water in the coils 512 a, 512 b, and the air iscooled.

The speed of the fan 510 and/or the flow rate or temperature of coolingwater flowing in the cooling coils 512 a, 512 b may be controlled inresponse to measured values. For example, the pumps driving the coolingliquid may be variable speed pumps that are controlled to maintain aparticular temperature in work space 506. Such control mechanisms may beused to maintain a constant temperature in workspace 506 or plenums 504a, 504 b and attic 505.

The workspace 506 air may then be drawn into racks 502 a, 502 b such asby fans mounted on the many trays that are mounted in racks 502 a, 502b. This air may be heated as it passes over the trays and through powersupplies running the computers on the trays, and may then enter thewarm-air plenums 504 a, 504 b. Each tray may have its own power supplyand fan, with the power supply at the back edge of the tray, and the fanattached to the back of the power supply. All of the fans may beconfigured or programmed to deliver air at a single common temperature,such as at a set 113° F. (45° C.). The process may then be continuouslyreadjusted as fan 510 captures and circulates the warm air.

Additional items may also be cooled using system 500. For example, room516 is provided with a self-contained fan-coil unit 514 which contains afan and a cooling coil. The unit 514 may operate, for example, inresponse to a thermostat provided in room 516. Room 516 may be, forexample, an office or other workspace ancillary to the main portions ofthe data center 501.

In addition, supplemental cooling may also be provided to room 516 ifnecessary. For example, a standard roof-top or similar air-conditioningunit (not shown) may be installed to provide particular cooling needs ona spot basis. As one example, system 500 may be designed to deliver 78°F. (25.56° C.) supply air to work space 506, and workers may prefer tohave an office in room 516 that is cooler. Thus, a dedicatedair-conditioning unit may be provided for the office. This unit may beoperated relatively efficiently, however, where its coverage is limitedto a relatively small area of a building or a relatively small part ofthe heat load from a building. Also, cooling units, such as chillers,may provide for supplemental cooling, though their size may be reducedsubstantially compared to if they were used to provide substantialcooling for the system 500.

Fresh air may be provided to the workspace 506 by various mechanisms.For example, a supplemental air-conditioning unit (not shown), such as astandard roof-top unit may be provided to supply necessary exchanges ofoutside air. Also, such a unit may serve to dehumidify the workspace 506for the limited latent loads in the system 500, such as humanperspiration. Alternatively, louvers may be provided from the outsideenvironment to the system 500, such as powered louvers to connect to thewarm air plenum 504 b. System 500 may be controlled to draw air throughthe plenums when environmental (outside) ambient humidity andtemperature are sufficiently low to permit cooling with outside air.Such louvers may also be ducted to fan 510, and warm air in plenums 504a, 504 b may simply be exhausted to atmosphere, so that the outside airdoes not mix with, and get diluted by, the warm air from the computers.Appropriate filtration may also be provided in the system, particularlywhere outside air is used.

Also, the workspace 506 may include heat loads other than the trays,such as from people in the space and lighting. Where the volume of airpassing through the various racks is very high and picks up a very largethermal load from multiple computers, the small additional load fromother sources may be negligible, apart from perhaps a small latent heatload caused by workers, which may be removed by a smaller auxiliary airconditioning unit as described above.

Cooling water may be provided from a cooling water circuit powered bypump 524. The cooling water circuit may be formed as a direct-return, orindirect-return, circuit, and may generally be a closed-loop system.Pump 524 may take any appropriate form, such as a standard centrifugalpump. Heat exchanger 522 may remove heat from the cooling water in thecircuit. Heat exchanger 522 may take any appropriate form, such as aplate-and-frame heat exchanger or a shell-and-tube heat exchanger.

Heat may be passed from the cooling water circuit to a condenser watercircuit that includes heat exchanger 522, pump 520, and cooling tower518. Pump 520 may also take any appropriate form, such as a centrifugalpump. Cooling tower 518 may be, for example, one or more forced drafttowers or induced draft towers. The cooling tower 518 may be considereda free cooling source, because it requires power only for movement ofthe water in the system and in some implementations the powering of afan to cause evaporation; it does not require operation of a compressorin a chiller or similar structure.

The cooling tower 518 may take a variety of forms, including as a hybridcooling tower. Such a tower may combine both the evaporative coolingstructures of a cooling tower with a water-to-water heat exchanger. As aresult, such a tower may be fit in a smaller face and be operated moremodularly than a standard cooling tower with separate heat exchanger.Additional advantage may be that hybrid towers may be run dry, asdiscussed above. In addition, hybrid towers may also better avoid thecreation of water plumes that may be viewed negatively by neighbors of afacility.

As shown, the fluid circuits may create an indirect water-sideeconomizer arrangement. This arrangement may be relatively energyefficient, in that the only energy needed to power it is the energy foroperating several pumps and fans. In addition, this system may berelatively inexpensive to implement, because pumps, fans, coolingtowers, and heat exchangers are relatively technologically simplestructures that are widely available in many forms. In addition, becausethe structures are relatively simple, repairs and maintenance may beless expensive and easier to complete. Such repairs may be possiblewithout the need for technicians with highly specialized knowledge.

Alternatively, direct free cooling may be employed, such as byeliminating heat exchanger 522, and routing cooling tower water(condenser water) directly to cooling coils 512 a, 512 b (not shown).Such an implementation may be more efficient, as it removes one heatexchanging step. However, such an implementation also causes water fromthe cooling tower 518 to be introduced into what would otherwise be aclosed system. As a result, the system in such an implementation may befilled with water that may contain bacteria, algae, and atmosphericcontaminants, and may also be filled with other contaminants in thewater. A hybrid tower, as discussed above, may provide similar benefitswithout the same detriments.

Control valve 526 is provided in the condenser water circuit to supplymake-up water to the circuit. Make-up water may generally be neededbecause cooling tower 518 operates by evaporating large amounts of waterfrom the circuit. The control valve 526 may be tied to a water levelsensor in cooling tower 518, or to a basin shared by multiple coolingtowers. When the water falls below a predetermined level, control valve526 may be caused to open and supply additional makeup water to thecircuit. A back-flow preventer (BFP) may also be provided in the make-upwater line to prevent flow of water back from cooling tower 518 to amain water system, which may cause contamination of such a water system.

Optionally, a separate chiller circuit may be provided. Operation ofsystem 500 may switch partially or entirely to this circuit during timesof extreme atmospheric ambient, e.g., hot and humid, conditions or timesof high heat load in the data center 501. Controlled mixing valves 534are provided for electronically switching to the chiller circuit, or forblending cooling from the chiller circuit with cooling from thecondenser circuit. Pump 528 may supply tower water to chiller 530, andpump 532 may supply chilled water, or cooling water, from chiller 530 tothe remainder of system 500. Chiller 530 may take any appropriate form,such as a centrifugal, reciprocating, or screw chiller, or an absorptionchiller.

The chiller circuit may be controlled to provide various appropriatetemperatures for cooling water. In some implementations, the chilledwater may be supplied exclusively to a cooling coil, while in others,the chilled water may be mixed, or blended, with water from heatexchanger 522, with common return water from a cooling coil to bothstructures. The chilled water may be supplied from chiller 530 attemperatures elevated from typical chilled water temperatures. Forexample, the chilled water may be supplied at temperatures of 55° F.(13° C.) to 65 to 70° F. (18 to 21° C.) or higher. The water may then bereturned at temperatures like those discussed below, such as 59 to 176°F. (15 to 80° C.). In this approach that uses sources in addition to, oras an alternative to, free cooling, increases in the supply temperatureof the chilled water can also result in substantial efficiencyimprovements for the system 500.

Pumps 520, 524, 528, 532, may be provided with variable speed drives.Such drives may be electronically controlled by a central control system503 to change the amount of water pumped by each pump in response tochanging set points or changing conditions in the system 500. Forexample, pump 524 may be controlled to maintain a particular temperaturein workspace 506, such as in response to signals from a thermostat orother sensor in workspace 506.

As illustrated, control system 503 may be communicably coupled (shownthrough dashed lines that represent wired or wireless communication) toone or more components of the system 500. Although shown as coupled tosome, but not all, of the components of system 500 (e.g., valves, pumps,fans, VFDs, motor controllers, and otherwise), the control system 503 istypically communicably coupled to all components that require automatedoperation. In some aspects, the control system 503 comprises one or moremicroprocessor-based secondary controllers (e.g., corresponding to andpossibly mounted near each component of system 500) as well as, in someaspects, a microprocessor-based main controller that communicates withand controls at least some of the secondary controllers. Each controllermay include hardware and software instructions stored in memory andexecutable by the processor in each controller. In some aspects, thecontrol system 503 may perform one or more processes described herein,such as, for instance, method 200 shown in FIGS. 2A-2B.

In operation, system 500 (and more particularly, control system 503) mayrespond to signals from various sensors placed in the system 500. Thesensors may include, for example, thermostats, humidistats, flowmeters,and other similar sensors. In one implementation, one or morethermostats may be provided in warm air plenums 504 a, 504 b, and one ormore thermostats may be placed in workspace 506. In addition, airpressure sensors may be located in workspace 506, and in warm airplenums 504 a, 504 b. The thermostats may be used to control the speedof associated pumps, so that if temperature begins to rise, the pumpsturn faster to provide additional cooling waters. Thermostats may alsobe used to control the speed of various items such as fan 510 tomaintain a set pressure differential between two spaces, such as attic505 and workspace 506, and to thereby maintain a particular airflowrate. Where mechanisms for increasing cooling, such as speeding theoperation of pumps, are no longer capable of keeping up with increasingloads, a control system may activate chiller 530 and associated pumps528, 532, and may modulate control valves 534 accordingly to provideadditional cooling.

Various values for temperature of the fluids in system 500 may be usedin the operation of system 500. In one exemplary implementation, thetemperature setpoint in warm air plenums 504 a, 504 b may be selected tobe at or near a maximum exit temperature for trays in racks 502 a, 502b. This maximum temperature may be selected, for example, to be a knownfailure temperature or a maximum specified operating temperature forcomponents in the trays, or may be a specified amount below such a knownfailure or specified operating temperature. In certain implementations,a temperature of 45° C. may be selected. In other implementations,temperatures of 25° C. to 125° C. may be selected. Higher temperaturesmay be particularly appropriate where alternative materials are used inthe components of the computers in the data center, such as hightemperature gate oxides and the like.

In one implementation, supply temperatures for cooling water may be 68°F. (20° C.), while return temperatures may be 104° F. (40° C.). In otherimplementations, temperatures of 50° F. to 84.20° F. or 104° F. (10° C.to 29° C. or 40° C.) may be selected for supply water, and 59° F. to176° F. (15° C. to 80° C.) for return water. Chilled water temperaturesmay be produced at much lower levels according to the specifications forthe particular selected chiller. Cooling tower water supply temperaturesmay be generally slightly above the wet bulb temperature under ambientatmospheric conditions, while cooling tower return water temperatureswill depend on the operation of the system 500.

Using these parameters and the parameters discussed above for enteringand exiting air, relatively narrow approach temperatures may be achievedwith the system 500. The approach temperature, in this example, is thedifference in temperature between the air leaving a coil and the waterentering a coil. The approach temperature will always be positivebecause the water entering the coil is the coldest water, and will startwarming up as it travels through the coil. As a result, the water may beappreciably warmer by the time it exits the coil, and as a result, airpassing through the coil near the water's exit point will be warmer thanair passing through the coil at the water's entrance point. Because eventhe most-cooled exiting air, at the cooling water's entrance point, willbe warmer than the entering water, the overall exiting air temperaturewill need to be at least somewhat warmer than the entering cooling watertemperature.

Keeping the approach temperature small permits a system to be run onfree, or evaporative, cooling for a larger portion of the year andreduces the size of a needed chiller, if any is needed at all. To lowerthe approach temperature, the cooling coils may be designed forcounterflow rather than for self-draining. In counter-flow, the warmestair flows near the warmest water and the coolest air exits near wherethe coolest water enters.

In certain implementations, the entering water temperature may be 64° F.(18° C.) and the exiting air temperature 64.4° F. (25° C.), as notedabove, for an approach temperature of 12.6° F. (7° C.). In otherimplementations, wider or narrower approach temperature may be selectedbased on economic considerations for an overall facility.

With a close approach temperature, the temperature of the cooled airexiting the coil will closely track the temperature of the cooling waterentering the coil. As a result, the air temperature can be maintained,generally regardless of load, by maintaining a constant watertemperature. In an evaporative cooling mode, a constant watertemperature may be maintained as the wet bulb temperature stays constant(or changes very slowly), and by blending warmer return water withsupply water as the wet bulb temperature falls. As such, active controlof the cooling air temperature can be avoided in certain situations, andcontrol may occur simply on the cooling water return and supplytemperatures. The air temperature may also be used as a check on thewater temperature, where the water temperature is the relevant controlparameter.

These general and specific aspects described above may be implementedusing a device, system or method, or any combinations of devices,systems, or methods. For example, a system of one or more computers canbe configured to perform particular actions by virtue of havingsoftware, firmware, hardware, or a combination of them installed on thesystem that in operation causes or cause the system to perform theactions. One or more computer programs can be configured to performparticular actions by virtue of including instructions that, whenexecuted by data processing apparatus, cause the apparatus to performthe actions. The details of one or more implementations are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages will be apparent from the description anddrawings, and from the claims.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made. For example, othermethods described herein besides those, or in addition to those,illustrated in FIGS. 2A-2B can be performed. Further, the illustratedsteps of method 200 can be performed in different orders, eitherconcurrently or serially. Further, steps can be performed in addition tothose illustrated in method 200, and some steps illustrated in method200 can be omitted without deviating from the present disclosure.Further, various combinations of the components described herein may beprovided for implementations of similar apparatuses. Accordingly, otherimplementations are within the scope of the present disclosure.

What is claimed is:
 1. A method for cooling a data center, the methodcomprising: circulating a cooling airflow to cool a plurality ofcomputing devices supported in racks in a data center; monitoring powerusage of the plurality of computing devices; determining that a firstportion of the plurality of computing devices are operating at a powerusage above a threshold power usage; and based on the determination:decreasing a flow rate of a portion of a cooling liquid in the datacenter that is circulated to cool a second portion of the plurality ofcomputing devices that are operating at a power usage below thethreshold power usage; based on decreasing the flow rate of the portionof the cooling liquid in the data center that is circulated to cool thesecond portion of the plurality of computing devices, increasing aportion of the cooling airflow circulated to cool the second portion ofthe plurality of computing devices such that a rate of heat removal fromthe second portion of the plurality of computing devices remainssubstantially constant; and rerouting at least some of the portion ofthe cooling liquid in the data center that is circulated to cool thesecond portion of the plurality of computing devices to increase a flowrate of a portion of the cooling liquid in the data center that iscirculated to cool the first portion of the plurality of computingdevices such that a rate of heat removal from the first portion of theplurality of computing devices increases.
 2. The method of claim 1,wherein monitoring power usage of the plurality of computing devicescomprises monitoring a utilization of each of the plurality of computingdevices.
 3. The method of claim 2, wherein the threshold power usagecorresponds to an adjustable threshold utilization that is a particularpercentage of a maximum utilization.
 4. The method of claim 3, whereinthe adjustable threshold utilization is about 70% of maximumutilization.
 5. The method of claim 3, wherein utilization comprises CPUutilization of the plurality of computing devices.
 6. The method ofclaim 1, further comprising: monitoring a temperature at or near thefirst portion of the plurality of computing devices; and subsequent toincreasing flow rate of a portion of the cooling liquid in the datacenter that is circulated to cool the first portion of the plurality ofcomputing devices, determining that the temperature remainssubstantially constant or that the temperature rises at or near thefirst portion of the plurality of computing devices.
 7. The method ofclaim 6, further comprising: based on the determination that thetemperature remains substantially constant or that the temperature risesat or near the first portion of the plurality of computing devices,increasing a portion of the cooling airflow circulated to cool the firstportion of the plurality of computing devices such that the rate of heatremoval from the first portion of the plurality of computing devicesfurther increases.
 8. The method of claim 1, further comprising:determining that the first portion of the plurality of computing devicesare operating at a power usage at or below the threshold power usage fora predetermined time duration, or that the first portion of theplurality of computing devices are operating at a power usage at aparticular setpoint below the threshold power usage.
 9. The method ofclaim 8, further comprising: based on the determination, decreasing theflow rate of the portion of the cooling liquid in the data center thatis circulated to cool the first portion of the plurality of computingdevices; and modulating an airflow circulated to cool the first portionof the plurality of computing devices to maintain the rate of heatremoval from the first portion of the plurality of computing devices.10. The method of claim 1, wherein the data center comprises a coolingcapacity that is less than a cooling load required to cool all of theplurality of computing devices operating at a maximum power draw. 11.The method of claim 10, further comprising: deploying an additionalplurality of computing devices supported in racks in the data center.12. The method of claim 1, wherein the first portion of the plurality ofcomputing devices and the second portion of the plurality of computingdevices are located in different racks positioned in the data center.13. The method of claim 12, wherein the different racks are located indifferent rows of racks positioned in the data center.
 14. The method ofclaim 1, wherein a cooling power capacity of the data center coolingliquid supply volume is less than a heat output from the plurality ofcomputing devices supported in racks in the data center operating at100% utilization.
 15. The method of claim 14, wherein a sum of thecooling power capacity of the data center cooling liquid supply volumeand a cooling power capacity of a total cooling airflow volumetric flowrate exceeds the heat output from the plurality of computing devicessupported in racks in the data center operating at 100% utilization. 16.A computer storage medium encoded with a computer program, the programcomprising instructions that when executed by one or more computerscause the one or more computers to perform operations comprising:controlling a circulation of a cooling airflow to cool a plurality ofcomputing devices supported in racks in a data center; monitoring powerusage of the plurality of computing devices; determining that a firstportion of the plurality of computing devices are operating at a powerusage above a threshold power usage; and based on the determination:controlling one or more flow control devices to decrease a flow rate ofa portion of a cooling liquid in the data center that is circulated tocool a second portion of the plurality of computing devices that areoperating at a power usage below the threshold power usage; based oncontrolling the one or more flow control devices to decrease the flowrate of the portion of the cooling liquid in the data center that iscirculated to cool the second portion of the plurality of computingdevices, controlling one or more fans to increase a portion of thecooling airflow circulated to cool the second portion of the pluralityof computing devices such that a rate of heat removal from the secondportion of the plurality of computing devices remains substantiallyconstant; and controlling one or more flow control devices to reroute atleast some of the portion of the cooling liquid in the data center thatis circulated to cool the second portion of the plurality of computingdevices to increase a flow rate of a portion of the cooling liquid inthe data center that is circulated to cool the first portion of theplurality of computing devices such that a rate of heat removal from thefirst portion of the plurality of computing devices increases.
 17. Thecomputer storage medium of claim 16, wherein monitoring power usage ofthe plurality of computing devices comprises monitoring a utilization ofeach of the plurality of computing devices.
 18. The computer storagemedium of claim 17, wherein the threshold power usage corresponds to anadjustable threshold utilization that is a particular percentage of amaximum utilization.
 19. A data center cooling management system,comprising: a plurality of rack-mounted computers; one or more coolingmodules positioned near the plurality of rack-mounted computers; and acontrol system comprising one or more sensors and one or more flowcontrol devices, the control system operable to perform operationscomprising: controlling one or more cooling modules to circulate acooling airflow to cool the plurality of rack-mounted computers;monitoring, with the one or more sensors, power usage of the pluralityof rack-mounted computers; determining, with the one or more sensors,that a first portion of the plurality of rack-mounted computers areoperating at a power usage above a threshold power usage; and based onthe determination: controlling the one or more flow control devices todecrease a flow rate of a portion of a cooling liquid in the data centerthat is circulated to cool a second portion of the plurality ofrack-mounted computers that are operating at a power usage below thethreshold power usage; based on controlling the one or more flow controldevices to decrease the flow rate of the portion of the cooling liquidin the data center that is circulated to cool the second portion of theplurality of computing devices, controlling the one or more coolingmodules to increase a portion of the cooling airflow circulated to coolthe second portion of the plurality of rack-mounted computers such thata rate of heat removal from the second portion of the plurality ofrack-mounted computers remains substantially constant; and controllingthe one or more flow control devices to circulate at least some of theportion of the cooling liquid in the data center that is circulated tocool the second portion of the plurality of rack-mounted computers toincrease a flow rate of a portion of the cooling liquid in the datacenter that is circulated to cool the first portion of the plurality ofrack-mounted computers such that a rate of heat removal from the firstportion of the plurality of rack-mounted computers increases.
 20. Thesystem of claim 19, wherein monitoring power usage of the plurality ofrack-mounted computers comprises monitoring a utilization of each of theplurality of rack-mounted computers.
 21. The system of claim 19, whereinthe operations further comprise: monitoring, with the one or moresensors, a temperature at or near the first portion of the plurality ofrack-mounted computers; and subsequent to increasing flow rate of aportion of the cooling liquid in the data center that is circulated tocool the first portion of the plurality of rack-mounted computers,determining that the temperature remains substantially constant or thatthe temperature rises at or near the first portion of the plurality ofrack-mounted computers.
 22. A method for cooling a data center, themethod comprising: circulating a first cooling medium to cool aplurality of rack-mounted computers; circulating a second cooling mediumto cool the plurality of rack-mounted computers; determining that afirst portion of the plurality of rack-mounted computers is operating ata power usage above a threshold power usage; based on the determinationthat the first portion of the plurality of rack-mounted computers isoperating at the power usage above the threshold power usage, adjustinga flow rate of at least one of the first or second cooling mediums tocool a second portion of the plurality of rack-mounted computers that isoperating at a power usage at or below the threshold power usage;rerouting a portion of the at least one of the first or second coolingmediums from being circulated to the second portion of the plurality ofrack-mounted computers to cool the first portion of the plurality ofrack-mounted computers; and augmenting another portion of the at leastone of the first or second cooling mediums used to cool the firstportion of the plurality of rack-mounted computers with the reroutedportion of the at least one of the first or second cooling mediums. 23.The method of claim 22, further comprising: determining that the secondportion of the plurality of rack-mounted computers is operating at apower usage below a threshold power usage.
 24. The method of claim 22,wherein the first cooling medium is a cooling airflow and the secondcooling medium is a cooling liquid.
 25. The method of claim 24, whereinadjusting at least one of a flow rate of the first or second coolingmediums to cool a second portion of the plurality of rack-mountedcomputers comprises: increasing a flow rate of the cooling airflow tocool the second portion of the plurality of rack-mounted computers; anddecreasing a flow rate of the cooling liquid to cool the second portionof the plurality of rack-mounted computers.
 26. The method of claim 22,wherein circulating a second cooling medium to cool the plurality ofrack-mounted computers comprises: circulating, to the data center, acooling liquid flow from one or more central plants that comprise acooling capacity less than a maximum cooling capacity required to coolthe plurality of rack-mounted computers operating at a maximum powerload.